\chapter{Evaluation of the Design Artifact}

\section{Introduction}
\vspace{-4mm}
Following the DS approach \parencite{Hevner2004}, the objective is to achieve relevance and rigor. On the one hand side, rigor was substantiated by the execution of a literature review, which  collects foundations and methodologies reflected in today's state of research. On the other hand, a business need of more stringent instruments to support SO modelling have been motivated in the introduction. While metric suites represent the rigor part of the design artifact, the relevant part is formed by the overall method with its five steps, which unifies rigor and relevance by a stringent method supporting the measurement of service granularity and its dependent parameters to provide a detection tool for design flaws in SO modelling. As a last step of the DS approach, the idea is to evaluate the designed artifact to make its utility, efficacy and quality transparent, outlined in the upcoming sections of this thesis.

\section{Evaluation Method}
\vspace{-4mm}
In general the chosen evaluation method is dependent on the content of the designed artifact. \textcite[p. 86]{Hevner2004} recommends multiple evaluation methodologies, which could be \textit{observational}, \textit{analytical}, \textit{experimental}, \textit{testing} or \textit{descriptive}. As this thesis describes an innovative approach to relate service granularity and its dependent parameters to service models within a SOA, empirical methods are more appropriate to evaluate this artifact, because the overall approach must be investigated for correctness. Worth mentioning thereby is, that sub-tasks of involved steps in this approach must be quantitatively tested.

\noindent
With respect of the setup of the evaluation method, two independent, qualitative interviews have been done. The collected data in the interviews have been recorded, whereas its transcripts are available in the Appendix (starting on page \pageref{transcripts}).

\begin{figure}[ht]
		\begin{center}
	    	\includegraphics[page=73, trim=0cm 11.80cm 9.44cm 0cm, keepaspectratio=true, scale=.6, clip=true]{graphics/approach}
			\caption[Overview of interview setup]{Overview of interview setup}
			\label{Overview of interview setup}
		\end{center}
\end{figure}

\noindent
Each interview, took about 1.5 hours. For the first 50-60 minutes the artifact was explained by means of a presentation. In general, the conducted interviews focused on the overall approach, whereas topics like evaluation of metrics have been explained and discussed, but not challenged by the interview partners, as firstly the time of the interviews was limited and secondly metrics must be quantitatively evaluated. However, the utility and efficacy of the application of metrics within SOA has been discussed. Afterwards an interview was conducted with the intention to figure out the utility, efficacy and quality of the designed artifact.

\section{Evaluation Results}
\vspace{-4mm}

\subsection{Utility of Artifact}
\vspace{-4mm}
Cutting services along their dimensions (reach, range and realm) is seen as a permanent issue to be resolved in SOA initiatives. Benefits of applying the designed method in SOA initiatives  have been acknowledged. To be more concrete, answering the question around cutting services in proper granularity with respect of the service's purpose can be supported by the method. While for small projects only some parts of the method may be relevant, whereas the entire method might be seen in the context of large projects.

\noindent
As the method strictly starts with the definition of the target model, which shapes out a framework for service provisioning, the method could help in this context to increase transparency between Business and IT. However, it could be observed, that Business may not have any grip on the target model at all. This leads to the fact, that IT may not have any vision about the project, as Business may not be in a position to communicate this. As a consequence, IT's perception may increase to see it just as an IT related matter. Methods, as designed in the artifact, could be an enabler to align Business and IT.

\noindent
Comparing the answers relating service models, the interview partner, engaged in large-scale SOA projects confirmed, that not all involved roles (including IT and Business) know details behind the service model. This could be related to various facts, one might be limited communication. Relating to the partner engaged with small SOA projects, related IT persons understand the service models. Utilizing this method in the context of large-scaled project could increase understandability and thus improve the knowledge of the intended target model and its used service models.

\noindent
Measurements require meta-information about services of architectural elements in a service inventory. As complexity of the SOA initiative is seen as one criteria requiring to do so, it could be argued that the overall approach may only be applied in complex SOA projects. However, following the results of the interviews, today's SOA organizations may not be equipped with tools, allowing the administration of services in a formal way. It has been observed that related understanding documents are stored in Wikis and spreadsheet lists. Making use of the proposed method would rigorously require the collection of fine-grained meta-information too. Anyhow, one way to collect meta-information was attested to come from orchestration platforms with the crucial disadvantage, that this would be after implementation and therefore of limited potential.

\noindent
Finally, it could be stated, that the designed artifact is seen as a potential candidate to be partly used in practice, whereas implicit knowledge from IT architects about best-practice approaches to design services must formalized and related to metrics. Afterwards the design artifact could be applied with its steps in similar manner. Worth mentioning thereby is, that only simple metrics are seen to be reused.
\subsection{Efficacy of Artifact}
\vspace{-4mm}
The proposed way of applying only particular metrics, which could be related to service models, was understood and seen as effective, as metrics are mostly applied on a service level, thus also limiting the problem domain, when it comes to finding solutions for design flaws in SO modelling.

\noindent
The proposed method may be relevant for complex SOA projects, to align involved stakeholder and establish consistency with previously defined SOA guidelines.

\noindent
The potential of using metrics in SO modelling is rather seen at large-scale project sides. Moreover, the interpretation of metrics and follow-up action are highlighted as difficult task and driven by situational constraints at the client side.

\noindent
Generally the efficacy is seen dependent on the willingness of IT architects to execute the method. Moreover, due to the fact, that metrics have apparently not been used so far in practice, it could be concluded, that further field tests are required to firstly prove that the existing metrics lead to results, which can be used in practice.
\subsection{Discussion of Results}
\vspace{-4mm}
Measuring service granularity and its dependent parameters within in a SOA stays in its infancy. This gets obvious by a reflection of the state of research and the conducted interviews.

\noindent
Although SOA is seen as an integral concept, covering Business and IT, establishing a common ground between Business and IT is rarely seen in practice. When considering  research contributions in the area of measuring service granularity, tools are required to bridge the gap, in order SOA initiatives can be supported in its long term endeavour. It could be observed, that SOA initiatives starting on a Greenfield may deliver in the first years many ``features''.  However, not at least due to organizational change and knowledge limitations, extending a SOA in the long-rung in sustainable manner gets difficult without supporting tools.

\noindent
The designed method is seen in larger-scaled and complex projects. The bureaucracy will be higher, due to the fact, that methodological communication is required. Particularly, aligning between involved stakeholders the target model, service models and last but not least the whole process around the service inventory takes further resources. However, such a method aims to achieve better quality in the long-run, thus decreasing cost for maintenance is expected. Due to higher transparency, stakeholders might be more engaged and regular reviews of the SOA could increase Business-IT alignment. As the method focuses on service granularity related to service models, IT resources may have lower entry-criteria, because a ``divide and conquer'' strategy is in place, allowing to grow inside the SOA, as criteria, which could be related to objectives, are defined on a rather fine-grained level.